Methods and systems for detection and estimation of compression noise

ABSTRACT

Systems and methods for estimation of compression noise in an image or in a video sequence are provided. Difference values in a direction are computed, and one or more statistical feature values for the difference values are computed in association with a plurality of offsets. At least one feature value is computed using the plurality of statistical feature values, and a compression-noise estimate may be determined in accordance with a criterion based on a comparison of the at least on feature value and an associated threshold and an additive noise estimate.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/056,326, entitled “Methods and Systems for Detection and Estimationof Compression Noise,” filed Oct. 17, 2013, scheduled to issue on Jun.7, 2016 as U.S. Pat. No. 9,361,707, which is a continuation-in-part ofU.S. patent application Ser. No. 12/882,763, entitled “Methods andSystems for Estimation of Compression Noise,” filed on Sep. 15, 2010,U.S. Pat. No. 8,588,535 and a continuation-in-part of U.S. patentapplication Ser. No. 12/882,865, entitled “Methods and Systems for NoiseReduction and Image Enhancement,” filed on Sep. 15, 2010, U.S. Pat. No.8,600,188 and which also claims the benefit of U.S. ProvisionalApplication No. 61/888,463, filed Oct. 8, 2013, and U.S. ProvisionalApplication No. 61/888,437, filed Oct. 8, 2013. Each of U.S. patentapplication Ser. No. 14/056,326, U.S. patent application Ser. No.12/882,763, U.S. patent application Ser. No. 12/882,865, U.S.Provisional Application No. 61/888,463 and U.S. Provisional ApplicationNo. 61/888,437, are hereby incorporated by reference herein, in theirentirety.

FIELD OF THE INVENTION

The present invention relates to image processing and to videoprocessing and, in particular, to methods and systems for detection andestimation of compression noise in an image or a video sequence.

BACKGROUND

The quality of a video sequence or of a still image may vary widelydepending on the source. For example, computer-generated content may beof the highest quality. Packaged media content, for example, Blu-raymedia, may be of relatively high quality. Over-the-air high-definition(HD) broadcast content may be of relatively intermediate quality, whilecontent distributed via cable and satellite, may be of a relativelylower quality. Internet protocol television (IPTV) and streamed contentmay be of relatively low quality. Methods and systems for video andimage enhancement that automatically adapt to image- or video-contentquality may be desirable. Accordingly, methods and systems for detectionand estimation of compression noise in an image or in a video sequencemay be desirable.

SUMMARY

Some embodiments of the present invention comprise methods and systemsfor detection of compression noise in a still image or in a videosequence.

According to a first aspect of the present invention, difference valuesin a first direction are computed, and a plurality of statisticalfeature values are computed in associated with a plurality of offsets inthe first direction. At least one feature value is computed using theplurality of first-direction statistical feature values. Afirst-direction compression-noise estimate may be determined inaccordance with a criterion based on a comparison of the at least onefeature value and an associated threshold and an additive noiseestimate.

According to a second aspect of the present invention, difference valuesin a second direction are computed, and a plurality of statisticalfeature values are computed in associated with a plurality of offsets inthe second direction. At least one feature value is computed using theplurality of second-direction statistical feature values. Asecond-direction compression-noise estimate may be determined inaccordance with a criterion based on a comparison of the at least onefeature value and an associated threshold and an additive noiseestimate. The first-direction compression-noise estimate and thesecond-direction compression-noise estimate may be combined.

According to a third aspect of the present invention, statistics basedon statistical feature values of first-direction differences andstatistics based on statistical feature values of second-directiondifferences may be used to estimate the first-direction compressionnoise and the second-direction compression noise, respectively. Theestimated first-direction compression noise and the estimatedsecond-direction compression noise may be combined to form a compressionnoise estimate.

According to a fourth aspect of the present invention, features computedfrom statistical feature values of first-direction differences andfeatures computed from statistical feature values of second-directiondifferences may be compared to associated thresholds for determining ifan image or a video sequence has been compressed.

According to a fifth aspect of the present invention, an edge-preservingfilter may be controlled based on the estimated compression noiseestimate.

According to a sixth aspect of the present invention, a bi-lateralfilter may be controlled based on the estimated compression noiseestimate.

The foregoing and other objectives, features, and advantages of theinvention will be more readily understood upon consideration of thefollowing detailed description of the invention taken in conjunctionwith the accompanying drawings.

BRIEF DESCRIPTION OF THE SEVERAL DRAWINGS

FIG. 1 is a picture depicting exemplary embodiments of the presentinvention comprising controlling the image separation effectuated by anedge-preserving filter using an estimate of the noise in the inputimage;

FIG. 2 is a picture depicting exemplary embodiments of the presentinvention comprising estimation of compression noise by accessing alook-up table;

FIG. 3 is an exemplary, original, decoded image frame;

FIG. 4 is an enhanced image resulting from processing, according toembodiments of the present invention, of the exemplary, original,decoded image frame shown in FIG. 3, wherein the noise estimate used tocontrol the bi-lateral filter parameter value was based only on sensor,thermal and grain noise;

FIG. 5 is an enhanced image resulting from processing, according toembodiments of the present invention, of the exemplary, original,decoded image frame shown in FIG. 3, wherein the noise estimate used tocontrol the bi-lateral filter parameter value was based on sensor,thermal and grain noise and compression noise;

FIG. 6 is an exemplary, original, decoded image frame associated with alightly compressed image;

FIG. 7 is an enhanced image resulting from processing, according toembodiments of the present invention, of the exemplary, original decodedimage frame shown in FIG. 6, wherein the noise estimate used to controlthe bi-lateral filter parameter value was based only on compressionnoise;

FIG. 8 is an enhanced image resulting from processing, according toembodiments of the present invention, of the exemplary, original decodedimage frame shown in FIG. 6, wherein the noise estimate used to controlthe bi-lateral filter parameter value was based on noise due to sensor,thermal and grain noise and compression noise;

FIG. 9 is a picture depicting exemplary embodiments of the presentinvention comprising estimation of compression noise directly from imagedata;

FIG. 10 is a chart showing exemplary embodiments of the presentinvention comprising estimation of compression noise directly from imagedata for a still image or a single frame in a video sequence;

FIG. 11 is a picture illustrating exemplary pixel locations fordetermination of horizontal differences according to embodiments of thepresent invention;

FIG. 12 is a chart showing exemplary embodiments of the presentinvention comprising estimation of compression noise directly from imagedata for a frame in a video sequence;

FIG. 13 is a picture depicting exemplary embodiments of the presentinvention comprising un-sharp masking;

FIG. 14 is a picture depicting exemplary embodiments of the presentinvention comprising edge-preserving filter parameter valuedetermination based on the level of sharpening associated with theun-sharp masking;

FIG. 15 is a picture depicting exemplary embodiments of the presentinvention comprising edge-preserving filter parameter valuedetermination based on the level of sharpening associated with theun-sharp masking and based on temporal smoothing;

FIG. 16 is a plot showing two exemplary look-up tables for associating asharpening factor with a multiplier;

FIG. 17 is a picture depicting exemplary embodiments of the presentinvention comprising edge-preserving filter parameter valuedetermination based on the level of sharpening associated with theun-sharp masking;

FIG. 18 is a picture depicting an exemplary embodiment of the presentinvention comprising edge-preserving filter parameter valuedetermination based on the level of sharpening associated with theun-sharp masking;

FIG. 19 is an exemplary, original noise image;

FIG. 20 shows a result of bi-lateral filtering, using a sigma rangeparameter value equal to the standard deviation of the noise, of theoriginal noise image shown in FIG. 19;

FIG. 21 shows a result of bi-lateral filtering, according to embodimentsof the present invention, of the original noise image shown in FIG. 19,where the bi-lateral filter parameter value was set based on the noisein the original noise image;

FIG. 22 shows a result of bi-lateral filtering, according to embodimentsof the present invention, of the original noise image shown in FIG. 19,where the bi-lateral filter parameter value was set based on the noisein the original noise image and the level of sharpening;

FIG. 23 shows an exemplary, original image;

FIG. 24 shows a result of bi-lateral filtering, according to embodimentsof the present invention, of the original image shown in FIG. 23, wherethe bi-lateral filter parameter value was set based on noise estimatedfrom the original image;

FIG. 25 shows a result of bi-lateral filtering, according to embodimentsof the present invention, of the original image shown in FIG. 23, wherethe bi-lateral filter parameter value was set based on noise estimatedfrom the original image and the level of sharpening;

FIG. 26 is a chart showing exemplary embodiments of the presentinvention comprising methods for estimating additive noise based on afirst significant peak in a histogram of statistical feature values ofimage patches associated with an image-channel image;

FIG. 27 is a chart showing exemplary embodiments of the presentinvention comprising methods for estimating compression noise inaccordance with a criterion based on at least one feature value computedfrom statistical feature values corresponding to offsets in a firstdirection in a difference image and an additive noise estimate;

FIG. 28 is a chart showing exemplary embodiments of the presentinvention comprising methods for estimating compression noise inaccordance with a first criterion based on at least one feature valuecomputed from statistical feature values corresponding to offsets in afirst direction in a first-direction difference image and an additivenoise estimate and a second criterion based on at least one featurevalue computed for statistical feature values corresponding to offsetsin a second direction in a second-direction difference image and theadditive noise estimate; and

FIG. 29 is a picture depicting exemplary embodiments of the presentinvention comprising a hierarchy of edge-preserving filters, wherein theimage separation effectuated by the hierarchy of edge-preserving filtersis controlled based on an estimate of the noise in the input image.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Embodiments of the present invention will be best understood byreference to the drawings, wherein like parts are designated by likenumerals throughout. The figures listed above are expressly incorporatedas part of this detailed description.

It will be readily understood that the components of the presentinvention, as generally described and illustrated in the figures herein,could be arranged and designed in a wide variety of differentconfigurations. Thus, the following more detailed description of theembodiments of the methods, systems and apparatus of the presentinvention is not intended to limit the scope of the invention but it ismerely representative of the presently preferred embodiments of theinvention.

Elements of embodiments of the present invention may be embodied inhardware, firmware and/or a non-transitory computer program productcomprising a computer-readable storage medium having instructions storedthereon/in which may be used to program a computing system. Whileexemplary embodiments revealed herein may only describe one of theseforms, it is to be understood that one skilled in the art would be ableto effectuate these elements in any of these forms while resting withinthe scope of the present invention.

Although the charts and diagrams in the figures may show a specificorder of execution, it is understood that the order of execution maydiffer from that which is depicted. For example, the order of executionof a plurality of actions depicted by a corresponding plurality ofblocks in a figure may be changed relative to the shown order. Also, asa further example, two or more actions depicted in blocks shown insuccession in a figure may be executed concurrently, or with partialconcurrence. It is understood by those with ordinary skill in the artthat a non-transitory computer program product comprising acomputer-readable storage medium having instructions stored thereon/inwhich may be used to program a computing system, hardware and/orfirmware may be created by one of ordinary skill in the art to carry outthe various logical functions described herein.

The quality of a video sequence or of a still image may vary widelydepending on the source. For example, computer-generated content may beof the highest quality. Packaged media content, for example, Blu-raymedia, may be of relatively high quality. Over-the-air high-definition(HD) broadcast content may be of relatively intermediate quality, whilecontent distributed via cable and satellite, may be of a relativelylower quality. Internet protocol television (IPTV) and streamed contentmay be of relatively low quality. Methods and systems for video andimage enhancement that automatically adapt to image- or video-contentquality may be desirable. Additionally, methods and systems forautomatic estimation of an additive noise in an image or in a videosequence may be desirable.

When there is little, or no, noise in an image, either a still image ora frame in a video sequence, the image may be sharpened aggressively,and small details in the image may not be attenuated. However, selectiveimage sharpening may be required in the presence of noise. Stronger andlarger-scale features, as defined relative to the degree of noise, maybe identified and enhanced, while less significant and smaller-scalefeatures may be unprocessed, or potentially attenuated to mitigate thenoise present in the image.

U.S. patent application Ser. No. 12/228,774, entitled “Image SharpeningTechnique,” filed Aug. 15, 2008, which is hereby incorporated byreference herein in its entirety, describes a technique for imagesharpening wherein two image channels are generated from an image: afirst channel which may include primarily texture information and asecond channel which may include primarily edge and flat-regioninformation. The first channel may be filtered to attenuate higherfrequency content of the image in the first channel, and the secondchannel may be filtered to sharpen the image in the second channel. Thefiltered first channel and the filtered second channel may be combinedto form an enhanced image associated with the input image.

In some embodiments of the present invention, an edge-preserving filtermay be used to separate an input image into two channels: a firstchannel which may comprise image content to be sharpened; and a secondchannel which may comprise image content to attenuate. In someembodiments of the present invention, the edge-preserving filter maycomprise a bi-lateral filter. In alternative embodiments, anotheredge-preserving filter may be used to perform the separation. In someembodiments, the assignment of the input-image content to the firstchannel and the second channel may be controlled by a parameter of theedge-preserving filter. For example, a range sigma parameter of abi-lateral filter may control the assignment of input-image content tothe first channel and the second channel in embodiments wherein theedge-preserving filter comprises a bi-lateral filter. In someembodiments, in order to maximize the amount of sharpening, the filterparameter value, for example, the range sigma parameter value in abi-lateral filter, may be set based on noise statistics associated withthe input image.

Some embodiments of the present invention may be understood in relationto FIG. 1. An input image 2 and a noise estimate 3 associated with theinput image 2 may be made available to an edge-preserving filter 4. Insome embodiments of the present invention, the edge-preserving filtermay comprise a bi-lateral filter. In alternative embodiments, anotheredge-preserving filter may be used to perform the separation. Theedge-preserving filter 4 may be used to divide the input image 2 intotwo channels 6, 10. A first channel 6 may correspond to significantfeatures in the input image 2, and the second channel 10, formed byremoving 8 the first channel 6 from the input image 2, may contain theresidual difference between the first channel 6 and the input image 2.The residual difference may comprise noise and texture data. The firstchannel 6 may be sharpened 12, and the second channel 10 may beattenuated 14. The sharpened channel 16 and the attenuated channel 18may be combined 20 to form an enhanced image 22. The value of a filterparameter of the edge-preserving filter 4 may control the assignment ofinput-image content to the first channel 6 and the second channel 10.The value of the filter parameter of the edge-preserving filter 4 may bebased on the noise estimate 3. In some embodiments of the presentinvention, the value of the filter parameter may be proportional to thenoise estimate 3. However, the value of the filter parameter may not beequal to zero. In some embodiments, when the noise estimate 3 indicatesthat the filter parameter should be set to zero, a small, pre-determinedvalue may be assigned to the filter parameter. In alternativeembodiments, when the noise estimate 3 indicates that the filterparameter should be set to zero, the edge-preserving filter 4 may passthe input image 2 directly through as the first channel 6, therebyeffectuating only sharpened data. In some embodiments of the presentinvention (not shown), the enhanced image 22 may be up-sampled to aresolution greater than that of the input image 2.

Multiple noise processes may appear in an image or video sequence. Atone extreme, an original image may be of very high quality, for example,computer generated imagery. However, other images, or video sequences,may be of lower quality, for example, a sequence or image may becorrupted, during the acquisition process by thermal, sensor, or othernoise. In some situations, film grain, or other analog noise, maycorrupt an image or video sequence. Furthermore, compression artifactsmay corrupt an image or video sequence. The degree of compressionartifacts may be related to the bit-rate associated with a videosequence or image. Exemplary bit-rates and quality levels may be highquality Blu-ray discs compressed at approximately 40 Mbps (Mega bits persecond), lower quality over-the-air transmissions compressed atapproximately 20 Mbps, further lower quality trans-coded cabletransmissions compressed at approximately 12 Mbps and lowest qualitysatellite and IPTV services compressed at less than 10 Mbps.

In some embodiments of the present invention described in relation toFIG. 2, an input bitstream 30, associated with a still image or a videosequence, may be decoded by an appropriate decoder 32. The image data 34may be made available, to an image-noise statistic estimator 38, fromthe decoder 32, and the decoded coding parameters 36 may be madeavailable, to a codec (coder/decoder)-noise statistic estimator 40, fromthe decoder 32. In alternative embodiments, the coding parameters 36 maybe made available to the codec-noise statistic estimator 40 throughmeta-data or other external means.

The image-noise statistic estimator 38 may analyze the image data 34 todetermine the amount of thermal, sensor, grain, or other image noisepresent in the image data 34. Many existing methods are known in the artfor estimating the image noise present in image data. In an exemplaryembodiment of the present invention, the image-noise statistic estimator38 may identify one, or more, substantially smooth regions in the imagedata 34 and may calculate the standard deviation of the pixel valueswithin each region. The maximum standard-deviation value may beassociated with an image-noise statistic 44.

The codec-noise statistic estimator 40 may receive coding parameters 36associated with the input bitstream 30. In some embodiments, the codingparameters 36 may comprise the quantization interval used for coding thecurrent slice or frame in a video sequence. In alternative embodiments,the coding parameters 36 may comprise the number of bits used torepresent the source data. The coding parameters 36 may be used in alook-up operation to determine the amount of noise due to compression.In some embodiments of the present invention, the codec type may be usedto select a look-up table from multiple, pre-computed look-up tables 42.The coding parameters 36 may be used as an index into the selectedtable, and the output may be a measure of the image noise due tocompression 46. The measurement provided by the look-up tables 42 may benormalized, either at the time the look-up tables 42 are created or in apost-processing operation (not shown) to have the same units ofmeasurement as the noise estimate 44 provided by the image-noisestatistic estimator 38.

In some embodiments of the present invention, a look-up table 42 may beconstructed by selecting multiple images and video sequences that arerepresentative of input data. The images and videos sequences may becompressed using a variety of codecs and codec settings, and each resultmay be subsequently decoded. The decoded results may be compared to theoriginal data, and the noise due to the compression system may becomputed, for example, the standard deviation of the error. Thiscomputation may be performed over all images and video sequences thatare compressed using the same codec and parameter settings, and theresult may be stored in the look-up table as the noise statistic for thecombination of codec and codec parameter settings. In some embodiments,if additional values are needed, the test image and video sequence datamay be compressed with the desired configuration settings. Inalternative embodiments, noise statistics may be interpolated fromvalues with a similar compression configuration.

The image-noise statistic 44 and the codec-noise statistic 46 may becombined by a combiner 48 to produce a final noise estimate 50, whichmay be made available to the edge-preserving filter to control thefilter parameter. In one embodiment of the present invention, themaximum of the image-noise statistic 44 and the codec-noise statistic 46may be assigned to the noise estimate 50. Alternative fusion methods forcombining 48 the image-noise statistic 44 and the codec-noise statistic46 may be used to produce a final noise estimate 50 which may be madeavailable to the edge-preserving filter, and the value of the filterparameter of the edge-preserving filter may be based on the noiseestimate. In some embodiments of the present invention, the value of thefilter parameter may be proportional to the noise estimate.

The effectiveness of some embodiments of the present invention may beillustrated in FIGS. 3-8. FIG. 3 depicts an exemplary, original, decodedimage 60 with two regions 62, 64 that exhibit coding artifacts showninside two white circles superimposed on the image 60. One region 62 isalong the road, and another region 64 is around a car. FIG. 4 shows anenhancement 70 of the decoded image 60 shown in FIG. 3. The enhancedimage 70 was generated using a bi-lateral filter using only an estimateof the image noise to select the range sigma parameter value of thebi-lateral filter. The coding artifacts are visibly enhanced, forexample, the regions 72, 74 that are shown in the two superimposed,white circles. However, the enhanced image 80 shown in FIG. 5, bycontrast, was enhanced according to embodiments of the present inventionwherein a noise estimate accounting for both the image noise, forexample, sensor, thermal, grain and other image noise, and thecompression noise in the image frame was used to select the range sigmaparameter value of the bi-lateral filter. Using this noise estimateproduces an enhanced image 80, but without amplifying the codingartifacts, for example, again examine the regions 82, 84 inside thesuperimposed, white circles.

A second exemplary, original, decoded image 90 is depicted in FIG. 6.Here, the image 90 is the result of light compression. Inspection ofFIG. 6 shows a significant amount of grain noise, which may become muchmore visible if enhanced. Shown in FIG. 7 is an enhanced image 100wherein the noise estimate controlling the parameter selection in thebi-lateral filter considered only the coding noise. As can be seen inFIG. 7, the enhanced image 100 contains amplified grain noise. However,the enhanced image 110 shown in FIG. 8 resulting from image enhancementaccording to embodiments of the present invention wherein the noiseestimate used to select the range sigma parameter for the bi-lateralfilter accounted for both the image noise, for example, sensor, thermal,grain and other noise, and the compression noise in an image frame. Ascan be seen from FIG. 8, the resulting image is enhanced but withoutamplifying grain noise artifacts.

Embodiments of the present invention described in relation to FIG. 2comprise codec-noise statistic estimation from the coding parameters. Inalternative embodiments described in relation to FIG. 9, the compressionnoise may be estimated from the decoded image data.

In some embodiments of the present invention, described in relation toFIG. 9, comprising a noise-estimation system 115, an input bitstream120, associated with a still image or a video sequence, may be decodedby a decoder 122. The image data 124 may be made available, from thedecoder 122, to an image-noise statistic estimator 126 and to acodec-noise statistic estimator 128. The image-noise statistic estimator126 may estimate an image-noise statistic 130 associated with the amountof image noise present in the image data 124, and the codec-noisestatistic estimator 128 may estimate a codec-noise statistic 132associated with the compression noise. The image-noise statistic 130 andthe codec-noise statistic 132 may be combined by a combiner 134 toproduce a final noise estimate 136 associated with the image data. Inone embodiment of the present invention, the maximum value of theimage-noise statistic 130 and the codec-noise statistic 132 may beassigned to the noise estimate 136. Alternative fusion methods forcombining 134 the image-noise statistic 130 and the codec-noisestatistic 132 may be used to produce a final noise estimate 136.

The image-noise statistic estimator 126 may analyze the image data 124to determine the amount of thermal, sensor, grain and other image noisepresent in the image data 124. Many existing methods are known in theart for estimating the image noise. In an exemplary embodiment of thepresent invention, the image-noise statistic estimator 126 may identifyone, or more, substantially smooth regions in the image data 124 and maycalculate the standard deviation of the pixel values within each region.The maximum standard-deviation value may be associated with theimage-noise statistic 130.

In some embodiments of the present invention, the codec-noise statisticestimator 128 may calculate an estimate for the codec noise 132according to FIG. 10. In these embodiments, a luminance image may becomputed 140 from an input RGB (Red Green Blue), or other color space,image. The luminance may be denoted I(r,c), where r and c may denote therow and column indices, respectively. A horizontal difference value ateach point may be computed 142, thereby producing a plurality ofhorizontal difference values, and a vertical difference value at eachpoint may be computed 144, thereby producing a plurality of verticaldifference values, according to:

D _(H)(r,c)=I(r,c+1)−I(r,c)

and

D _(V)(r,c)=I(r+1,c)−I(r,c),

respectively, where D_(H) (r,c) may denote a horizontal difference valueat pixel location (r,c) and D_(V) (r,c) may denote a vertical differencevalue at pixel location (r,c).

The standard deviation of the horizontal differences at a plurality ofhorizontal offsets may be computed 146, and the standard deviation ofthe vertical differences at a plurality of vertical offsets may becomputed 148. In some embodiments of the present invention, astandard-deviation value may be calculated for each offset within acoding block, thereby producing a plurality of horizontalstandard-deviation values and a plurality of vertical standard-deviationvalues. The number of offsets may be determined by the structure of theimage, or video, codec and any processing or scaling of the decoded datathat may be performed prior to estimating the compression noise.

In an exemplary embodiment comprising the use of eight offsets, thestandard-deviation values of the horizontal differences may becalculated 146, for an image of width denoted NumCols and height denotedNumRows, according to:

${{{Mean}_{H}\lbrack i\rbrack} = {\sum\limits_{r = 0}^{NumRows}\; {\sum\limits_{c = 0}^{{NumCols}/8}\; \frac{D_{H}\left( {r,{{8 \cdot c} + i}} \right)}{{NumRows} \cdot \left( {{NumCols}/8} \right)}}}},{i = {0\mspace{14mu} \ldots \mspace{14mu} 7}}$${{{Mean}\; {2_{H}\lbrack i\rbrack}} = {\sum\limits_{r = 0}^{NumRows}\; {\sum\limits_{c = 0}^{{NumCols}/8}\; \frac{{D_{H}\left( {r,{{8 \cdot c} + i}} \right)}^{2}}{{NumRows} \cdot \left( {{NumCols}/8} \right)}}}},{i = {0\mspace{14mu} \ldots \mspace{14mu} 7}}$STD_(H)[i] = sqrt(Mean 2_(H)[i] − Mean_(H)[i]²), i = 0  …  7.

FIG. 11 depicts an exemplary portion of an image 160 with pixellocations shown as squares. The pixel locations shown in cross hatch,for example 162, may be the locations used for computing the horizontalstandard deviation associated with an offset of zero, STD_(H) [0], andthe pixel locations shown in white, for example 164, may be thelocations used for computing the horizontal standard deviationassociated with an offset of two, STD_(H) [2].

The standard-deviation values of the vertical differences may besimilarly calculated 148 according to:

${{{Mean}_{V}\lbrack i\rbrack} = {\sum\limits_{r = 0}^{{NumRows}/8}\; {\sum\limits_{c = 0}^{NumCols}\; \frac{D_{V}\left( {{{8 \cdot r} + i},c} \right)}{\left( {{NumRows}/8} \right) \cdot {NumCols}}}}},{i = {0\mspace{14mu} \ldots \mspace{14mu} 7}}$${{{Mean}\; {2_{V}\lbrack i\rbrack}} = {\sum\limits_{r = 0}^{{NumRows}/8}\; {\sum\limits_{c = 0}^{NumCols}\; \frac{{D_{V}\left( {{{8 \cdot r} + i},c} \right)}^{2}}{\left( {{NumRows}/8} \right) \cdot {NumCols}}}}},{i = {0\mspace{14mu} \ldots \mspace{14mu} 7}}$STD_(V)[i] = sqrt(Mean 2_(V)[i] − Mean_(V)[i]²), i = 0  …  7.

Referring again to FIG. 10, statistics may be computed 150 for thehorizontal standard-deviation values, and statistics may be computed 152for the vertical standard-deviation values. The statistics computed 150for the horizontal standard-deviation values may be referred to ashorizontal statistics and the statistics computed 152 for the verticalstandard-deviation values may be referred to as vertical statistics. Insome embodiments of the present invention, the average of the horizontalstandard-deviation values and the maximum horizontal standard-deviationvalue may be computed 150, and the average of the verticalstandard-deviation values and the maximum vertical standard-deviationvalue may be computed 152. These values may be computed according to:

$\begin{matrix}{{STD}_{H\_ {MEAN}} = {\sum\limits_{i = 0}^{7}\; \frac{{STD}_{H}\lbrack i\rbrack}{8}}} \\{{{STD}_{H\_ {MAX}} = {\max \left( {{STD}_{H}\lbrack i\rbrack} \right)}},} \\{i = {0\mspace{14mu} \ldots \mspace{14mu} 7}} \\{{STD}_{V\_ {MEAN}} = {\sum\limits_{i = 0}^{7}\; \frac{{STD}_{V}\lbrack i\rbrack}{8}}} \\{{{STD}_{V\_ {MAX}} = {\max \left( {{STD}_{V}\lbrack i\rbrack} \right)}},} \\{{i = {0\mspace{14mu} \ldots \mspace{14mu} 7}},}\end{matrix}$

respectively. In alternative embodiments, other statistical values maybe calculated, for example, the median, the skew, the kurtosis and otherstatistical measures.

The computed statistics associated with the horizontalstandard-deviation values may be combined 154 to form an estimate of thehorizontal compression noise. The computed statistics associated withthe vertical standard-deviation values may be combined 156 to form anestimate of the vertical compression noise. In an exemplary embodimentof the present invention, an estimate of the horizontal compressionnoise may be calculated according to a weighted average given by:

Noise_(Compression) _(_) _(H)=4.64·STD _(H) _(_) _(MAX)−4.26·STD _(H)_(_) _(MEAN)+0.58,

and an estimate of the vertical compression noise may be calculatedaccording to a weighted average given by:

Noise_(Compression) _(_) _(V)=4.64·STD _(V) _(_) _(MAX)−4.26·STD _(V)_(_) _(MEAN)+0.58,

where the values 4.64, 4.26 and 0.58 are exemplary weighting parameters.In alternative embodiments, other weighting values may be used.

The estimate for the horizontal compression noise and the estimate forthe vertical compression noise may be combined 158 to form a singlecompression noise estimate. Any data fusion method known in the art maybe used to combine the estimate for the horizontal compression noise andthe estimate for the vertical compression noise. In some embodiments ofthe present invention, the compression-noise estimate may be determinedaccording to:

Noise_(Compression)=max(Noise_(Compression) _(_)_(H),Noise_(Compression) _(_) _(V)).

In alternative embodiments of the present invention, the codec-noisestatistic estimator 128 may calculate an estimate for the codec noise132 according to FIG. 12. In these embodiments, the input image may be aframe in a video sequence, and past frames may be considered indetermining the codec-noise estimate. In these embodiments, a luminanceimage may be computed 170 from an input RGB, or other color space,image, and the luminance may be denoted I(r,c), where r and c may denotethe row and column indices, respectively. A horizontal difference valueat each point may be computed 172, thereby producing a plurality ofhorizontal difference values, and a vertical difference value at eachpoint may be computed 174, thereby producing a plurality of verticaldifference values, according to:

D _(H)(r,c)=I(r,c+1)−I(r,c)

and

D _(V)(r,c)=I(r+1,c)−I(r,c),

respectively, where D_(H)(r,c) may denote a horizontal difference valueat pixel location (r,c) and D_(V)(r,c) may denote a vertical differencevalue at pixel location (r,c).

The standard deviation of the horizontal differences at a plurality ofhorizontal offsets may be computed 176, and the standard deviation ofthe vertical differences at a plurality of vertical offsets may becomputed 178. In some embodiments of the present invention, astandard-deviation value may be calculated for each offset within acoding block, thereby producing a plurality of horizontalstandard-deviation values and a plurality of vertical standard-deviationvalues. The number of offsets may be determined by the structure of thevideo codec and any processing or scaling of the decoded data that maybe performed prior to estimating the compression noise.

In an exemplary embodiment comprising the use of eight offsets, thestandard-deviation values of the horizontal differences may becalculated 176 according to:

$\begin{matrix}{{{{Mean}_{H}\lbrack i\rbrack} = {\sum\limits_{r = 0}^{NumRows}\; {\sum\limits_{c = 0}^{{NumCols}/8}\; \frac{D_{H}\left( {r,{{8 \cdot c} + i}} \right)}{{NumRows} \cdot \left( {{NumCols}/8} \right)}}}},} \\{i = {0\mspace{14mu} \ldots \mspace{14mu} 7}} \\{{{{Mean}\; {2_{H}\lbrack i\rbrack}} = {\sum\limits_{r = 0}^{NumRows}\; {\sum\limits_{c = 0}^{{NumCols}/8}\; \frac{{D_{H}\left( {r,{{8 \cdot c} + i}} \right)}^{2}}{{NumRows} \cdot \left( {{NumCols}/8} \right)}}}},} \\{i = {0\mspace{14mu} \ldots \mspace{14mu} 7}} \\{{{{STD}_{H}\lbrack i\rbrack} = {{sqrt}\left( {{{Mean}\; {2_{H}\lbrack i\rbrack}} - {{Mean}_{H}\lbrack i\rbrack}^{2}} \right)}},} \\{i = {0\mspace{14mu} \ldots \mspace{14mu} 7.}}\end{matrix}$

The standard-deviation values of the vertical differences may besimilarly calculated 178 according to:

$\begin{matrix}{{{{Mean}_{V}\lbrack i\rbrack} = {\sum\limits_{r = 0}^{{NumRows}/8}\; {\sum\limits_{c = 0}^{NumCols}\; \frac{D_{V}\left( {{{8 \cdot r} + i},c} \right)}{\left( {{NumRows}/8} \right) \cdot {NumCols}}}}},} \\{i = {0\mspace{14mu} \ldots \mspace{14mu} 7}} \\{{{{Mean}\; {2_{V}\lbrack i\rbrack}} = {\sum\limits_{r = 0}^{{NumRows}/8}\; {\sum\limits_{c = 0}^{NumCols}\; \frac{{D_{V}\left( {{{8 \cdot r} + i},c} \right)}^{2}}{\left( {{NumRows}/8} \right) \cdot {NumCols}}}}},} \\{i = {0\mspace{14mu} \ldots \mspace{14mu} 7}} \\{{{{STD}_{V}\lbrack i\rbrack} = {{sqrt}\left( {{{Mean}\; {2_{V}\lbrack i\rbrack}} - {{Mean}_{V}\lbrack i\rbrack}^{2}} \right)}},} \\{i = {0\mspace{14mu} \ldots \mspace{14mu} 7.}}\end{matrix}$

Statistics may be computed 180 for the horizontal standard-deviationvalues, and statistics may be computed 182 for the verticalstandard-deviation values. In some embodiments of the present invention,the average of the horizontal standard-deviation values and the maximumhorizontal standard-deviation value may be computed 180, and the averageof the vertical standard-deviation values and the maximum verticalstandard-deviation value may be computed 182. These values may becomputed according to:

$\begin{matrix}{{STD}_{H\_ {MEAN}} = {\sum\limits_{i = 0}^{7}\; \frac{{STD}_{H}\lbrack i\rbrack}{8}}} \\{{{STD}_{H\_ {MAX}} = {\max \left( {{STD}_{H}\lbrack i\rbrack} \right)}},} \\{i = {0\mspace{14mu} \ldots \mspace{14mu} 7}} \\{{STD}_{V\_ {MEAN}} = {\sum\limits_{i = 0}^{7}\; \frac{{STD}_{V}\lbrack i\rbrack}{8}}} \\{{{STD}_{V\_ {MAX}} = {\max \left( {{STD}_{V}\lbrack i\rbrack} \right)}},} \\{{i = {0\mspace{14mu} \ldots \mspace{14mu} 7}},}\end{matrix}$

respectively. In alternative embodiments, other statistical values maybe calculated, for example, the median, the skew, the kurtosis and otherstatistical measures.

The computed statistics associated with the horizontalstandard-deviation values may be combined 184 to form an estimate of thehorizontal compression noise. The computed statistics associated withthe vertical standard-deviation values may be combined 186 to form anestimate of the vertical compression noise. In an exemplary embodimentof the present invention, an estimate of the horizontal compressionnoise may be calculated according to a weighted average given by:

Noise_(Compression) _(_) _(H)=4.64·STD _(H) _(_) _(MAX)−4.26·STD _(H)_(_) _(MEAN)+0.58,

and an estimate of the vertical compression noise may be calculatedaccording to a weighted average given by:

Noise_(Compression) _(_) _(V)=4.64·STD _(V) _(_) _(MAX)−4.26·STD _(V)_(_) _(MEAN)+0.58,

where the values 4.64, 4.26 and 0.58 are exemplary weighting parameters.In alternative embodiments, other weighting values may be used.

The location of block boundaries in the current image frame may beestimated 188, 190 using the statistics previously computed. In anexemplary embodiment, a horizontal-block boundary location may beestimated 188 using the maximum horizontal standard-deviation valueaccording to:

Block_(H)[frame_num]=i, where STD _(H) _(_) _(MAX) ==STD _(H) [i],

where frame_num may denote a time index associated with the currentframe. A vertical block-boundary location may be estimated 190 by themaximum vertical standard-deviation value according to:

Block_(V)[frame_num]=i, where STD _(V) _(_) _(MAX) ==STD _(V) [i],

where frame_num may denote a time index associated with the currentframe.

The number of unique horizontal block-boundary locations and the numberof unique vertical block-boundary locations in a temporal block may bedetermined 192, 194 by counting the number of unique values forBlock_(H) [i] and Block_(V)[i], respectively, where i is an index withvalues from frame_num to frame_num−N and N is a constant. If the numberof unique values for the horizontal direction is above a threshold, thenthe estimate for the horizontal compression noise may be set equal tozero, and if the number of unique values for the vertical direction isabove a threshold, then the estimate for the vertical compression noisemay be set equal to zero.

The estimate for the horizontal compression noise and the estimate forthe vertical compression noise may be combined 196 to form a singlecompression noise estimate. Any data fusion method known in the art maybe used to combine the estimate for the horizontal compression noise andthe estimate for the vertical compression noise. In some embodiments ofthe present invention, the compression-noise estimate may be determinedaccording to:

Noise_(Compression)=max(Noise_(Compression) _(_)_(H),Noise_(Compression) _(_) _(V)).

In some embodiments of the present invention, the parameters of anedge-preserving filter may be adjusted based on a sharpening value usedin a sharpening filter. In some exemplary embodiments, theedge-preserving filter may comprise a bi-lateral filter. In exemplaryembodiments of the present invention shown in FIG. 13 and FIG. 14, anedge-preserving filter 204 may be used to separate an input image 200into two channels: a first channel 206 which may comprise image contentto be sharpened by un-sharp masking (USM) 212; and a second channel 210which may comprise image content to attenuate. The level of sharpeningassociated with the un-sharp masking 212 may be controlled by the valueof the USM alpha 213 set for the USM filter 212. In these embodiments, afilter parameter value 203 of the edge-preserving filter 204 may controlthe assignment of input-image content to the first channel 206 and thesecond channel 210. In some embodiments of the present invention whereinthe edge-preserving filter comprises a bi-lateral filter, the filterparameter value may be the range sigma parameter value associated withthe bi-lateral filter.

In some embodiments, in order to maximize the amount of sharpening, thefilter parameter 203 may be set based on the noise statistics associatedwith the input image 200, and additionally, in order to mitigate noiseamplification due to sharpening, the filter parameter 203 may be setbased on the sharpening value 213 used in the sharpening filter 212.

An input image 200 and a filter parameter value 203 associated with theinput image may be made available to an edge-preserving filter 204. Theedge-preserving filter 204 may be used to divide the input image 200into two channels 206, 210. The first channel 206 may correspond tosignificant features in the input image 200, and the second channel 210,formed by removing 208 the first channel 206 from the input image 200,may contain the residual difference between the first channel 206 andthe input image 200. The residual difference 210 may comprise noise andtexture data. The first channel 206 may be sharpened using un-sharpmasking 212, and the second channel 210 may be attenuated 214 inaccordance with noise control 215. The sharpened channel 216 and theattenuated channel 218 may be combined 220 to form an enhanced image222. The filter parameter value 203 of the edge-preserving filter 204may control the assignment of input-image content to the first channel206 and the second channel 210. In some embodiments of the presentinvention (not shown), the enhanced image 222 may be up-sampled to aresolution greater than that of the input image 200.

Referring to FIG. 14, the input image 200 may be received by a noiseestimator 230 which may generate a noise estimate 231 based on anestimate of the image noise and the codec noise present in the inputimage 200. In some embodiments of the present invention, the noiseestimate 231 may be determined according to the noise-estimationembodiments of the present invention described herein. The noiseestimate 231 may be adjusted by a USM sharpening compensator 232 toaccount for the sharpening value, and the resulting sharpening-adjustedvalue 233 may be temporally filtered 234 to reduce noise in the controlprocess. The output of the temporal filter 234 may be the filterparameter value 203 provided to the edge-preserving filter 204.

In some embodiments of the present invention described in relation toFIG. 14 and FIG. 15, the noise estimator 230 may comprise an image-noisestatistic estimator 250 and a codec-noise statistic estimator 252 forestimating image noise and codec noise associated with an input image200, respectively. The noise estimator 230 may comprise a combiner 254for combining the image noise estimated by the image-noise statisticestimator 250 and the codec noise codec-noise statistic estimator 252.The noise estimate 231 may be provided to the USM sharpening compensator232. In some embodiments, the noise estimate 231 may be converted to astandard-deviation value in the noise estimator 230. In alternativeembodiments, the noise estimate may be converted to a standard-deviationvalue in the USM sharpening compensator 232.

The USM sharpening compensator 232 may adjust the noise estimate 231 bya value associated with the level of sharpening 213. In some embodimentsof the present invention, a sharpening compensator controller 256 mayuse the USM alpha value 213 as an index into a look-up table 258 todetermine a multiplier 257 which may be used to multiplicatively adjust260 the standard-deviation-value noise estimate 231. FIG. 16 depicts twoplots 280, 282 each associated with an exemplary look-up table derivedto maintain a constant noise level with increasing sharpening factor.

In addition to adjusting the noise estimate 231 by the value 257associated with the level of sharpening 213, the noise estimate 231 maybe multiplicatively 260 adjusted by a value associated with controlparameter, which may be denoted M, that controls how quickly theestimated parameter values may change as a function of time. In someembodiments the multiplicative adjustment 260 may be 1−M. In someembodiments of the present invention, a smaller value for M maycorrespond to a faster change, while a larger value for M may correspondto a slower change. In some embodiments of the present invention, M maybe set to 0.5. A filter parameter value 263 associated with a previousframe may be retrieved from storage 262 and multiplied 264 by thecontrol parameter M. The results of the two multiplications 260, 264 maybe added 266 to form the edge-preserving filter parameter 203 for thenext frame.

In alternative embodiments described in relation to FIG. 17, anedge-preserving filter parameter may be determined for filtering of astill image, or a video frame. An input image 300 may be received by anoise estimator 302 which may generate a noise estimate 303 based on anestimate of the image noise and the codec noise present in the inputimage 300. The noise estimate 303 may be determined according to thenoise-estimation embodiments of the present invention described herein.The noise estimate 303 may be adjusted by a USM sharpening compensator304 to account for the sharpening value 305, and the resultingsharpening-adjusted value 306 may be the filter parameter value 306provided to the edge-preserving filter 204.

In some embodiments of the present invention described in relation toFIG. 17 and FIG. 18, the noise estimator 302 may comprise an image-noisestatistic estimator 310 and a codec-noise statistic estimator 312 forestimating image noise and codec noise associated with an input image300, respectively. The noise estimator 302 may comprise a combiner 314for combining the image noise estimated by the image-noise statisticestimator 310 and the codec noise codec-noise statistic estimator 312.The noise estimate 303 may be provided to the USM sharpening compensator304. In some embodiments, the noise estimate 303 may be converted to astandard-deviation value in the noise estimator 302. In alternativeembodiments, the noise estimate 303 may be converted to astandard-deviation value in the USM sharpening compensator 304.

The USM sharpening compensator 304 may adjust the noise estimate 303 bya value associated with the level of sharpening 305. In some embodimentsof the present invention, a sharpening compensator controller 316 mayuse the USM alpha value 305 as an index into a look-up table 318 todetermine a multiplier 319 which may be used to multiplicatively adjust320 the standard-deviation-value noise estimate 303 to produce asharpening-adjusted edge-preserving filter parameter value 306. FIG. 16depicts two plots 280, 282 each associated with an exemplary look-uptable derived to maintain a constant noise level with increasingsharpening factor.

The effectiveness of embodiments of the present invention may beillustrated in FIGS. 19-25. FIG. 19 depicts an exemplary, original,synthetic noise target 350. FIG. 20 shows the result 352 of bi-lateralfiltering, using a sigma range parameter value equal to the standarddeviation of the noise, of the original image 350 shown in FIG. 19. Anenhanced image 354 is shown in FIG. 21. This enhanced image 354 exhibitssignificant amplification of the image noise. FIG. 22 depicts anenhanced image 356 enhanced according to embodiments of the presentinvention in which the bi-lateral filter sigma range parameter is setaccording to the noise statistics of the image in addition to accountingfor the sharpening strength of the sharpening filter. The noise in FIG.22 is similar to the noise in FIG. 20, but the degree of enhancement isdifferent.

A second exemplary, original image 360 is depicted in FIG. 23. Asuperimposed, white circle 361 is shown on the image 360 in a regionwherein the noise in the image 360 is visible. FIG. 24 depicts an image362 which results from bi-lateral filtering according to embodiments ofthe present invention with a range sigma parameter that has been setaccounting for the noise, but not the sharpening process. Inspection ofthe region inside the overlaid white circle 363 shows that the noise isamplified relative to the input image 360. FIG. 25 depicts an image 364which results from bi-lateral filtering according to embodiments of thepresent invention with a range sigma parameter that has been setaccounting for both noise and the sharpening process. Inspection of theregion inside the overlaid white circle 365 shows that the noise is notamplified relative to the original image 360, but the desired increasein sharpening of image content is visible.

When an image or video sequence has a large amount of texture, manysystems and methods for estimating non-codec noise may fail due to astrong correlation between texture and an additive-noise estimatecomputed based on standard deviations of image patches, andconsequently, over-smoothing may occur.

Some embodiments of the present invention described in relation to FIG.26 comprise a method 400 for estimating additive noise in a still imageor in a frame of a video sequence.

In these embodiments, a statistical feature value may be computed 402for each image patch in a plurality of image patches of an image-channelimage associated with an input image. Exemplary image-channel images area luminance image, a chrominance image, an R-component image of an RGBimage, a G-component image of an RGB image, a B-component image of anRGB image, a chroma-channel image in an image color space comprising achroma channel, a combination of image channels, and other image-channelimages. The image-channel image may be denoted I(x, y), where x and ymay denote the horizontal and vertical indices, respectively, and I(x,y) may denote the image-channel image value at an image locationcorresponding to (x, y).

In some embodiments of the present invention, each patch may be an N×Npatch. In some embodiments of the present invention, the patches of theimage-channel image may be overlapping patches. In some embodiments ofthe present invention, the plurality of images patches of theimage-channel image may comprise all possible N×N patches in theimage-channel image. In some embodiments of the present invention, N maybe five pixels.

In alternative embodiments of the present invention, the plurality ofimage patches may comprise all image patches, whereat the value of apixel, on which an image patch is centered, meets a first criterion. Insome exemplary embodiments, the plurality of image patches may compriseall N×N image patches, whereat the center value of the image patch meetsa first criterion. In some exemplary embodiments, the plurality of imagepatches may comprise all N×N image patches, whereat the center value ofthe image patch corresponds to a luminance value in the range [luma_low,luma_high], where luma_low is a first luminance value denoting alower-limit on a range of luminance values and luma_high is a secondluminance value denoting an upper-limit on the range of luminancevalues. Exemplary ranges are [16, 235] and [16, 240].

In alternative embodiments of the present invention, the plurality ofimage patches may comprise all image patches for which a characteristicof an image patch meets a first criterion.

In some embodiments of the present invention, the image-channel imagemay be computed from an input image, for example, from an image in anRGB, or other, color space. In some embodiments of the presentinvention, the image-channel image may be a combination of imagechannels.

In some embodiments of the present invention, the statistical featurevalue may be a standard-deviation value. In alternative embodiments, thestatistical feature value may be another statistic, for example, thevariance, the sum of absolute differences, and other statistical values.In embodiments wherein the statistical feature value is astandard-deviation value, the standard-deviation value, which may bedenoted STD_(P), of an image patch, which may be denoted P and for whichPx and Py may denote the sets of x and y indices, respectively,corresponding to the pixel locations, in the image-channel image, withinpatch P, may be computed according to:

$\begin{matrix}{{Mean}_{P} = {\sum\limits_{Px}^{\;}\; {\sum\limits_{Py}^{\;}\; \frac{I\left( {x,y} \right)}{N \cdot N}}}} \\{{{Mean}\; 2_{P}} = {\sum\limits_{Px}^{\;}\; {\sum\limits_{Py}^{\;}\; \frac{{I\left( {x,y} \right)}^{2}}{N \cdot N}}}} \\{{{STD}_{P} = {{sqrt}\left( {{{Mean}\; 2_{P}} - {Mean}_{P}^{2}} \right)}},}\end{matrix}$

where sqrt (•) may denote the square-root function.

A histogram of the statistical feature values computed for theimage-channel patches may be formed 404. In some embodiments of thepresent invention, the histogram may be formed by accumulating thestatistical feature values for all image patches. In alternativeembodiments of the present invention, the histogram may be formed byaccumulating the statistical feature values for all image patches intoan initial histogram and smoothing the initial histogram. In some ofthese embodiments, the initial histogram may be smoothed using amoving-window averaging function. In alternative embodiments, analternative smoothing method known in the art may be used to smooth theinitial histogram.

The histogram may be searched 406 for a first significant peak, wherethe first significant peak is a significant peak whereat the binlocation of the significant peak corresponds to the smallest statisticalfeature value compared to that of other significant peaks. In someembodiments of the present invention, a histogram peak may be identifiedusing a peak-search window function at each histogram-bin location. Alocation may be considered a peak location if the histogram countassociated with the leftmost histogram-bin location in the peak-searchwindow is greater than the histogram count associated with the rightmosthistogram-bin location in the peak-search window. In some embodiments ofthe present invention, a peak may be considered significant when thenumber of histogram counts within a significance-determination windowcentered at the peak location meets a first criterion. In someembodiments of the present invention, a peak may be considered asignificant peak if the number of histogram counts within thesignificance-determination window is greater than 66 percent of thenumber of pixels in the luminance image. A person of ordinary skill inthe art will recognize the existence of many methods and systems forlocating a significant peak within a histogram. The above-describedmethods are for illustration purposes and should not be considered alimitation of embodiments of the present invention.

A determination may be made 408 as to whether or not a first significantpeak has been found. If no significant peak has been found 410, then thenoise estimate may be set 412 to a default value. In some embodiments ofthe present invention wherein the noise estimation is for a frame of avideo sequence, the default value may be a noise estimate from aprevious frame. In some embodiments, the noise estimate may be from theimmediately previous frame. In alternative embodiments, the defaultvalue may be a last reliable noise estimate from a previous frame. Insome embodiments of the present invention wherein the noise estimationis for a still image, the default value may be set to a pre-determinedvalue.

The noise estimate may be labeled 414 with a label indicating a “lowconfidence” in the accuracy and validity of the noise estimate.

The noise-estimation process 400 may end 416.

If a first significant peak is identified 418, then a peak-width measuremay be computed 420. In some embodiments of the present invention, thepeak-width measure may be computed by determining the position withinthe histogram, prior to the first-significant-peak location, whereat thehistogram count falls below a predetermined threshold and subtractingthat location from the first-significant-peak location. In someembodiments of the present invention, the predetermined threshold may berelated to the histogram count at the first-significant-peak location.In some embodiments of the present invention, the threshold may be 50%of the histogram count at the first-significant-peak location.

A determination may be made 422 as to whether or not the peak-widthmeasure indicates a broad peak. In some embodiments of the presentinvention, the peak-width measure may be compared to a width threshold,and if the peak-width measure is greater than the width threshold, thenthe peak may be considered a broad peak. If the first significant peakis considered a broad peak 424, then the noise estimate may be set 412to a default value. In some embodiments of the present invention whereinthe noise estimation is for a frame of a video sequence, the defaultvalue may be a noise estimate from a previous frame. In someembodiments, the noise estimate may be from the immediately previousframe. In alternative embodiments, the default value may be a lastreliable noise estimate from a previous frame. In some embodiments ofthe present invention wherein the noise estimation is for a still image,the default value may be set to a pre-determined value.

The noise estimate may be labeled 414 with a label indicating a “lowconfidence” in the accuracy and validity of the noise estimate.

The noise-estimation process 400 may end 416.

If the first significant peak is not considered a broad peak 426, then anoise estimate may be computed 428 based on the first-significant-peaklocation and the peak-width measure, and the noise-estimation process400 may end 416.

In some embodiments of the present invention, the noise estimate may becomputed 428 by combining the statistical feature value corresponding tothe histogram bin, prior to the first-significant-peak location, whereatthe histogram count falls below the predetermined threshold with thestatistical feature value corresponding to the first-significant-peaklocation. In some embodiments of the present invention, the noiseestimate may be computed 428 by subtracting the statistical featurevalue corresponding to the histogram bin, prior to thefirst-significant-peak location, whereat the histogram count falls belowthe predetermined threshold from the statistical feature valuecorresponding to the first-significant-peak location and combining thedifference with the statistical feature value corresponding to thefirst-significant-peak location. In alternative embodiments, a factor ofthe difference between the statistical feature value corresponding tothe histogram bin, prior to the first-significant-peak location, whereatthe histogram count falls below the predetermined threshold and thestatistical feature value corresponding to the first-significant-peaklocation may be combined with the statistical feature valuecorresponding to the first-significant-peak location. In someembodiments of the present invention, combining the values may compriseadding the values together.

Some embodiments of the present invention described in relation to FIG.27 may comprise a method 500 for detecting and estimating compressionnoise in an image or in a video sequence.

In these embodiments, a difference value, in a first direction, at eachpoint in an image-channel image may be computed 502, thereby producing aplurality of first-direction difference values. Exemplary image-channelimages are a luminance image, a chrominance image, an R-component imageof an RGB image, a G-component image of an RGB image, a B-componentimage of an RGB image, a chroma-channel image in an image color spacecomprising a chroma channel, a combination of image channels, and otherimage-channel images.

In some embodiments of the present invention, the image-channel imagemay be computed from an input image, for example, from an image in anRGB, or other, color space.

A statistical feature value, for example, a standard-deviation value, avariance value, a sum-of-absolute-differences value and otherstatistical feature value, of the first-direction difference values atan offset in the first direction may be computed 504 for each of theoffsets in a plurality of first-direction offsets, thereby producing aplurality of first-direction statistical feature values.

In some embodiments of the present invention, a statistical featurevalue may be calculated for each offset, in the first direction, withina coding block. The number of offsets may be determined by the structureof the image, or video, codec and any processing or scaling of thedecoded data that may be performed prior to estimating the compressionnoise.

One, or more, first-direction feature values may be computed 506 usingthe computed statistical feature values for the first-direction offsets,also considered the first-direction statistical feature values anddenoted SF_(D1) [i], wherein i may correspond to a first-directionoffset value.

In some embodiments of the present invention, a first-directionnear-peak-value feature value may be computed by determining the numberof first-direction statistical feature values that are close to themaximum first-direction statistical feature value. This first-directionnear-peak-value feature value may be referred to as the NearPeakD1 valueand may be determined by counting the number of first-directionstatistical feature values satisfying:

SF _(D1) [i]≧T _(D1) _(_) _(CLOSE) SF _(D1) _(_) _(MAX),

where

${{SF}_{D\; 1{\_ {MAX}}} = {\max\limits_{i}\left( {{SF}_{D\; 1}\lbrack i\rbrack} \right)}},$

i corresponds to a first-direction offset, max (•) denotes the maximumfunction and T_(D1) _(_) _(CLOSE) is a closeness threshold. In someembodiments of the present invention, T_(D1) _(_) _(CLOSE)=0.99.

In some embodiments of the present invention, a first-direction variancefeature value may be the value of the variance of the first-directionstatistical feature values, which may be denoted VAR_(D1).

In an exemplary embodiment comprising the use of ND1 offsets, thevariance of the first-direction statistical feature values may becomputed according to:

${SF}_{D\; 1{\_ {MEAN}}} = {\sum\limits_{i = 0}^{{{ND}\; 1} - 1}\; \frac{{SF}_{D\; 1}\lbrack i\rbrack}{{ND}\; 1}}$${{SF}\; 2_{D\; 1{\_ {MEAN}}}} = {\sum\limits_{i = 0}^{{{ND}\; 1} - 1}\; \frac{\left( {{SF}_{D\; 1}\lbrack i\rbrack} \right)^{2}}{{ND}\; 1}}$VAR_(D 1) = SF 2_(D 1_MEAN) − SF_(D 1_MEAN)².

In some embodiments of the present invention, a first-directionblock-boundaries feature may relate to an estimate of the locations offirst-direction block boundaries in a plurality of video frames in avideo sequence.

In an exemplary embodiment, a first-direction block boundary locationmay be estimated using the maximum first-direction statistical featurevalue according to:

${{{Block}_{D\; 1}\lbrack{frame\_ num}\rbrack} = i},{{{where}\mspace{14mu} {\max\limits_{i}\left( {{SF}_{D\; 1}\lbrack i\rbrack} \right)}}{{SF}_{D\; 1}\lbrack i\rbrack}},$

where frame_num may denote a time index associated with a video frame.

The number of unique first-direction block-boundary locations, which maybe referred to as NumBlockLocsD1, may be determined by counting thenumber of unique values for Block_(D1) [j], where j is an index withvalues from frame_num to frame_num−N and N is a constant.

The first-direction feature values computed using the first-directionstatistical feature values may each be compared 508 to an associatedthreshold, wherein the threshold may be selected to delineate betweenfeature values indicative of an image, or video frame, compressed in thefirst direction, and those indicative of an image, or video frame,uncompressed in the first direction.

An additive noise estimate may be received 510. A determination may bemade 512 to whether or not the first-direction feature values and theadditive noise estimate meet a first-direction no-compression criterionbased on the first-direction feature value/threshold comparisons and thereceived additive noise estimate.

If the first-direction no-compression criterion is met 514, then afirst-direction compression-noise estimate may be set 516 to an“ignore-estimate” value indicating that the first-directioncompression-noise estimate may be ignored. In some embodiments of thepresent invention, the first-direction compression-noise estimate may beset 516 to zero to indicate that the first-direction compression-noiseestimate may be ignored.

If the first-direction no-compression criterion is not met 518, thenone, or more, statistics may be computed 520 from the first-directionstatistical feature values.

In some embodiments of the present invention, the average of thefirst-direction statistical feature values and the maximumfirst-direction statistical feature value may be computed. These valuesmay be computed according to:

$\begin{matrix}{{SF}_{D\; 1{\_ {MEAN}}} = {\sum\limits_{i = 0}^{{{ND}\; 1} - 1}\; \frac{{SF}_{D\; 1}\lbrack i\rbrack}{{ND}\; 1}}} \\{{{SF}_{D\; 1{\_ {MAX}}} = {\max\limits_{i}\left( {{SF}_{D\; 1}\lbrack i\rbrack} \right)}},{i = {{0\mspace{14mu} \ldots \mspace{14mu} {ND}\; 1} - 1}},}\end{matrix}$

respectively, wherein ND1 is the number of first-direction offsets. Inalternative embodiments, other statistical values may be calculated, forexample, the median, the skew, the kurtosis and other statisticalmeasures.

The computed statistics associated with the first-direction statisticalfeature values may be combined 522 to form an estimate of thefirst-direction compression noise.

In an exemplary embodiment of the present invention, an estimate of thefirst-direction compression noise may be calculated according to aweighted average given by:

Noise_(Compression) _(_) _(D1)=4.64·SF _(D1) _(_) _(MAX)−4.26·SF _(D1)_(_) _(MEAN)+0.58,

where the values 4.64, 4.26 and 0.58 are exemplary weighting parameters.In alternative embodiments, other weighting values may be used.

For example, in an alternative exemplary embodiment of the presentinvention, an estimate of the first-direction compression noise may becalculated according to a weighted average given by:

Noise_(Compression) _(_) _(D1)=1.6005·SF _(D1) _(_) _(MAX)−1.0764·SF_(D1) _(_) _(MEAN)+2.7259.

In an exemplary embodiment of the present invention, a first-directionnear-peak-value feature value, NearPeakD1, may be compared to afirst-direction near-peak-value threshold, and if the first-directionnear-peak-value feature value is above the first-directionnear-peak-value threshold, then the first-direction near-peak-valuefeature value may indicate an image, or video frame uncompressed in thefirst direction. In some exemplary embodiments, the first-directionnear-peak-value threshold may be 50% of the number of first-directionoffsets. In some exemplary embodiments wherein the statistical featurevalue is the standard-deviation value, if NearPeakD1>2, then nocompression in the first direction may be indicated.

In an exemplary embodiment of the present invention, a first-directionvariance feature value, VAR_(D1), may be compared to a first-directionvariance threshold, and if VAR_(D1) is below the first-directionvariance threshold, then the first-direction variance feature value mayindicate an image, or video frame, uncompressed in the first direction.In some exemplary embodiments wherein the statistical feature value isthe standard-deviation value, if VAR_(D1)<0.6, then no compression inthe first direction may be indicated.

In an exemplary embodiment of the present invention, a first-directionblock-boundaries feature value, NumBlockLocsD1, may be compared to afirst-direction block-boundaries threshold, and if NumBlockLocsD1 isabove the first-direction block-boundaries threshold, then thefirst-direction block-boundaries feature value may indicate an image, orvideo frame, uncompressed in the first direction. In some exemplaryembodiments wherein the statistical feature value is thestandard-deviation value, if NumBlockLocsD1>2, when five temporal framesare examined, then no compression in the first direction may beindicated.

In an exemplary embodiment of the present invention, if the receivedadditive noise estimate is labeled a reliable estimate, then theindication is uncompressed image, or video frame.

In an exemplary embodiment of the present invention, if the receivedadditive noise estimate is greater than an additive-noise-estimatethreshold, then the indication is uncompressed image, or video frame.

In some exemplary embodiments of the present invention, when afirst-direction near-peak-value feature value indicates nofirst-direction compression OR a first-direction variance feature valueindicates no first-direction compression OR a third first-directionblock-boundaries feature value indicates no first-direction compressionOR the additive-noise estimate is reliable OR is greater than anassociated threshold value, then the first-direction compression-noiseestimate may be set to an “ignore-estimate” value indicating that thefirst-direction compression-noise estimate may be ignored. Otherwise,the first-direction compression-noise estimate may be computed bycombining one, or more, statistics computed using the first-directionstatistical feature values.

In some embodiments of the present invention, the first direction may bethe horizontal direction. In alternative embodiments of the presentinvention, the first direction may be the vertical direction.

In alternative embodiments 600 of the present invention described inrelation to FIG. 28, the process for estimation of compression noise ina first direction additionally may be performed in a second direction.

In these embodiments, a difference value, in a first direction, at eachpoint in an image-channel image may be computed 602, thereby producing aplurality of first-direction difference values, and a difference value,in a second direction, at each point in the image-channel may becomputed 622, thereby producing a plurality of second-directiondifference values.

In some exemplary embodiments of the present invention, a firstdirection may be a horizontal direction and a second direction may be avertical direction. In these embodiments, horizontal difference valuesand vertical difference values may be computed 602, 622 according to:

D _(H)(r,c)=I(r,c+1)−I(r,c)

and

D _(V)(r,c)=I(r+1,c)−I(r,c),

respectively, where I(r,c) may denote the image-channel image, r and cmay denote the row and column indices, respectively, D_(H) (r,c) maydenote a horizontal difference value at pixel location (r,c) and D_(V)(r,c) may denote a vertical difference value at pixel location (r,c).

In some embodiments of the present invention, the image-channel imagemay be computed from an input image, for example, from an image in anRGB, or other, color space.

A statistical feature value, for example, a standard-deviation value, avariance value, a sum-of-absolute-differences value and otherstatistical feature value, of the first-direction difference values at afirst-direction offset may be computed 604 for each of thefirst-direction offsets in a plurality of first-direction offsets,thereby producing a plurality of first-direction statistical featurevalues, and a statistical feature value, for example, astandard-deviation value, a variance value, asum-of-absolute-differences value and other statistical feature value,of the second-direction difference values at a second-direction offsetmay be computed 624 for each of the second-direction offsets in aplurality of second-direction offsets, thereby producing a plurality ofsecond-direction statistical feature values.

In some embodiments of the present invention, a statistical featurevalue may be calculated for each offset within a coding block, therebyproducing a plurality of horizontal statistical feature values and aplurality of vertical statistical feature values. The number of offsetsmay be determined by the structure of the image, or video, codec and anyprocessing or scaling of the decoded data that may be performed prior toestimating the compression noise.

In some embodiments of the present invention, the number of offsets inthe first direction may be the same as the number of offsets in thesecond direction. In some of these embodiments, the offset values may bethe same in each direction. In alternative of these embodiments, theoffset values may be different in each direction.

In alternative embodiments of the present invention, the number ofoffsets in the first direction may be different than the number ofoffsets in the second direction.

In an exemplary embodiment comprising the use of eight offsets andwherein the first direction is the horizontal direction and thestatistical feature value in the horizontal direction is thestandard-deviation value and the second direction is the verticaldirection and the statistical feature value in the vertical direction isthe standard-deviation value, the standard-deviation values of thehorizontal differences may be calculated 604, for an image of widthdenoted NumCols and height denoted NumRows, according to:

${{{Mean}_{H}\lbrack i\rbrack} = {\sum\limits_{r = 0}^{NumRows}\; {\sum\limits_{c = 0}^{{NumCols}/8}\; \frac{D_{H}\left( {r,{{8 \cdot c} + i}} \right)}{{NumRows} \cdot \left( {{NumCols}/8} \right)}}}},{i = {0\mspace{14mu} \ldots \mspace{14mu} 7}}$${{{Mean}\; {2_{H}\lbrack i\rbrack}} = {\sum\limits_{r = 0}^{NumRows}\; {\sum\limits_{c = 0}^{{NumCols}/8}\; \frac{{D_{H}\left( {r,{{8 \cdot c} + i}} \right)}^{2}}{{NumRows} \cdot \left( {{NumCols}/8} \right)}}}},{i = {0\mspace{14mu} \ldots \mspace{14mu} 7}},{{{STD}_{H}\lbrack i\rbrack} = {{sqrt}\left( {{{Mean}\; {2_{H}\lbrack i\rbrack}} - {{Mean}_{H}\lbrack i\rbrack}^{2}} \right)}},{i = {0\mspace{14mu} \ldots \mspace{14mu} 7}}$

where i denotes a horizontal offset and STD_(H) [i] denotes a horizontalstandard-deviation value associated with the horizontal offset denotedby i.

Correspondingly, in this exemplary embodiment, the standard-deviationvalues of the vertical differences may be calculated 624, for an imageof width denoted NumCols and height denoted NumRows, according to:

${{{Mean}_{V}\lbrack i\rbrack} = {\sum\limits_{r = 0}^{{NumRows}/8}\; {\sum\limits_{c = 0}^{NumCols}\; \frac{D_{V}\left( {{{8 \cdot r} + i},c} \right)}{\left( {{NumRows}/8} \right) \cdot {NumCols}}}}},{i = {0\mspace{14mu} \ldots \mspace{14mu} 7}}$${{{Mean}\; {2_{V}\lbrack i\rbrack}} = {\sum\limits_{r = 0}^{{NumRows}/8}\; {\sum\limits_{c = 0}^{NumCols}\; \frac{{D_{V}\left( {{{8 \cdot r} + i},c} \right)}^{2}}{\left( {{NumRows}/8} \right) \cdot {NumCols}}}}},{i = {0\mspace{14mu} \ldots \mspace{14mu} 7}},{{{STD}_{V}\lbrack i\rbrack} = {{sqrt}\left( {{{Mean}\; {2_{V}\lbrack i\rbrack}} - {{Mean}_{V}\lbrack i\rbrack}^{2}} \right)}},{i = {0\mspace{14mu} \ldots \mspace{14mu} 7}}$

where i denotes a horizontal offset and STD_(H) [i] denotes a horizontalstandard-deviation value associated with the horizontal offset denotedby i.

FIG. 11 depicts an exemplary portion of an image 160 with pixellocations shown as squares. The pixel locations shown in cross hatch,for example 162, may illustrate the locations used for computing thehorizontal standard-deviation value associated with an offset of zero,STD_(H) [0], and the pixel locations shown in white, for example 164,may illustrate the locations used for computing the horizontalstandard-deviation value associated with an offset of two, STD_(H) [2].

One, or more, first-direction features values may be computed 606 usingthe computed statistical feature values for the first-direction offsets,also considered the first-direction statistical feature values anddenoted SF_(D1) [i] where i=0, . . . , ND1−1 and ND1 is the number ofoffsets in the first direction, and one, or more, second-directionfeature values may be computed 626 using the computed statisticalfeature values for the second-direction offsets, also considered thesecond-direction statistical feature values and denoted SF_(D2)[i] wherei=0, . . . , ND2−1 and ND2 is the number of offsets in the seconddirection.

In some embodiments of the present invention, a first-directionnear-peak-value feature value may be computed by determining the numberof first-direction statistical feature values that are close to themaximum first-direction statistical feature value. This first-directionnear-peak-value feature value may be referred to as the NearPeakD1 valueand may be determined by counting the number of first-directionstatistical feature values satisfying:

SF _(D1) [i]≧T _(D1) _(_) _(CLOSE) SF _(D1) _(_) _(MAX),

where

${{SF}_{D\; 1{\_ {MAX}}} = {\max\limits_{i}\left( {{SF}_{D\; 1}\lbrack i\rbrack} \right)}},$

i corresponds to a first-direction offset, max(•) denotes the maximumfunction and T_(D1) _(_) _(CLOSE) is a closeness threshold. In someembodiments of the present invention T_(D1) _(_) _(CLOSE)=0.99.

In some embodiments of the present invention, a second-directionnear-peak-value feature value may be computed by determining the numberof second-direction statistical feature values that are close to themaximum second-direction statistical feature value. Thissecond-direction near-peak-value feature value may be referred to as theNearPeakD2 value and may be determined by counting the number ofsecond-direction standard-deviation values satisfying:

SF _(D2) [i]≧T _(D2) _(_) _(CLOSE) SF _(D2) _(_) _(MAX),

where

${{SF}_{D\; 2{\_ {MAX}}} = {\max\limits_{i}\left( {{SF}_{D\; 2}\lbrack i\rbrack} \right)}},$

i corresponds to a second-direction offset, max(•) denotes the maximumfunction and T_(D2) _(_) _(CLOSE) is a closeness threshold. In someembodiments of the present invention T_(D2) _(_) _(CLOSE)=0.99.

In some embodiments of the present invention, a first-direction variancefeature value may be the value of the variance of the first-directionstatistical feature values, which may be denoted VAR_(D1).

In an exemplary embodiment comprising the use of ND1 offsets, thevariance of the first-direction statistical feature values may becomputed according to:

$\begin{matrix}{{SF}_{D\; 1{\_ {MEAN}}} = {\sum\limits_{i = 0}^{{{ND}\; 1} - 1}\; \frac{{SF}_{D\; 1}\lbrack i\rbrack}{{ND}\; 1}}} \\{{{SF}\; 2_{D\; 1{\_ {MEAN}}}} = {\sum\limits_{i = 0}^{{{ND}\; 1} - 1}\; \frac{\left( {{SF}_{D\; 1}\lbrack i\rbrack} \right)^{2}}{{ND}\; 1}}} \\{{VAR}_{D\; 1} = {{{SF}\; 2_{D\; 1{\_ {MEAN}}}} - {{SF}_{D{1\_}{MEAN}}^{2}.}}}\end{matrix}$

In some embodiments of the present invention, a second-directionvariance feature value may be the value of the variance of thesecond-direction statistical feature values, which may be denotedVAR_(D2).

In an exemplary embodiment comprising the use of ND2 offsets, thevariance of the second-direction statistical feature values may becomputed according to:

$\begin{matrix}{{SF}_{D{2\_}{MEAN}} = {\sum\limits_{i = 0}^{{{ND}\; 2} - 1}\; \frac{{SF}_{D\; 2}\lbrack i\rbrack}{{ND}\; 2}}} \\{{{SF}\; 2_{D\; 2{\_ {MEAN}}}} = {\sum\limits_{i = 0}^{{{ND}\; 2} - 1}\; \frac{\left( {{SF}_{D\; 2}\lbrack i\rbrack} \right)^{2}}{{ND}\; 2}}} \\{{VAR}_{D\; 2} = {{{SF}\; 2_{D\; 2{\_ {MEAN}}}} - {{SF}_{D{2\_}{MEAN}}^{2}.}}}\end{matrix}$

In an exemplary embodiment comprising the use of eight offsets, thevariance of the horizontal statistical feature values may be computedaccording to:

${SF}_{H\_ {MEAN}} = {\sum\limits_{i = 0}^{7}\; \frac{{SF}_{H}\lbrack i\rbrack}{8}}$${{SF}\; 2_{H\_ {MEAN}}} = {\sum\limits_{i = 0}^{7}\; \frac{\left( {{SF}_{H}\lbrack i\rbrack} \right)^{2}}{8}}$VAR_(H) = SF 2_(H_MEAN) − SF_(H_MEAN)²

and the variance of the vertical statistical feature values maybecomputed according to:

$\begin{matrix}{{SF}_{V\_ {MEAN}} = {\sum\limits_{i = 0}^{7}\; \frac{{SF}_{V}\lbrack i\rbrack}{8}}} \\{{{SF}\; 2_{V\_ {MEAN}}} = {\sum\limits_{i = 0}^{7}\; \frac{\left( {{SF}_{V}\lbrack i\rbrack} \right)^{2}}{8}}} \\{{VAR}_{V} = {{{SF}\; 2_{V\_ {MEAN}}} - {{SF}_{V\_ {MEAN}}^{2}.}}}\end{matrix}$

In some embodiments of the present invention, a first-directionblock-boundaries feature may relate to an estimate of the locations offirst-direction block boundaries in a plurality of video frames in avideo sequence.

In an exemplary embodiment, a first-direction block boundary locationmay be estimated using the maximum first-direction statistical featurevalue according to:

${{{Block}_{D\; 1}\lbrack{frame\_ num}\rbrack} = i},{{{where}\mspace{14mu} {\max\limits_{i}\left( {{SF}_{D\; 1}\lbrack i\rbrack} \right)}}{{SF}_{D\; 1}\lbrack i\rbrack}},$

where frame_num may denote a time index associated with a video frame.

The number of unique first-direction block-boundary locations, which maybe referred to as NumBlockLocsD1, may be determined by counting thenumber of unique values for Block_(D1) [j], where j is an index withvalues from frame_num to frame_num−N and N is a constant.

In some embodiments of the present invention, a second-directionblock-boundaries feature may relate to an estimate of the locations ofsecond-direction block boundaries in a plurality of video frames in avideo sequence.

In an exemplary embodiment, a second-direction block boundary locationmay be estimated using the maximum second-direction statistical featurevalue according to:

${{{Block}_{D\; 2}\lbrack{frame\_ num}\rbrack} = i},{{{where}\mspace{14mu} {\max\limits_{i}\left( {{SF}_{D\; 2}\lbrack i\rbrack} \right)}} = {{SF}_{D\; 2}\lbrack i\rbrack}},$

where frame_num may denote a time index associated with a video frame.

The number of unique second-direction block-boundary locations, whichmay be referred to as NumBlockLocsD2, may be determined by counting thenumber of unique values for Block_(D2) [j], where j is an index withvalues from frame_num to frame_num−N and N is a constant.

The first-direction feature values computed using the first-directionstatistical feature values may each be compared 608 to an associatedthreshold, wherein the threshold may be selected to delineate betweenfeature values indicative of an image, or video frame, compressed in thefirst direction and those indicative of an image, or video frame,uncompressed image in the first direction. The second-direction featurevalues computed using the vertical statistical feature values may eachbe compared 628 to an associated threshold, wherein the threshold may beselected to delineate between feature values indicative of an image, orvideo frame, compressed in the second direction and those indicative ofan image, or video frame, uncompressed in the second direction.

An additive noise estimate may be received 610, 630. A determination maybe made 612 to whether or not the first-direction feature values and theadditive noise estimate meet a first-direction no-compression criterionbased on the first-direction feature value/threshold comparisons and thereceived additive noise estimate. A determination may be made 632 towhether or not the second-direction feature values and the additivenoise estimate meet a second-direction no-compression criterion, basedon the second-direction feature value/threshold comparisons and thereceived additive noise estimate.

If the first-direction no-compression criterion is met 614, then afirst-direction compression-noise estimate may be set 616 to an“ignore-estimate” value indicating that the first-directioncompression-noise estimate may be ignored. In some embodiments of thepresent invention, the first-direction compression-noise estimate may beset 616 to zero to indicate that the first-direction compression-noiseestimate may be ignored. If the second-direction no-compressioncriterion is met 634, then a second-direction compression-noise estimatemay be set 636 to an “ignore-estimate” value indicating that thesecond-direction compression-noise estimate may be ignored. In someembodiments of the present invention, the first-directioncompression-noise estimate may be set 636 to zero to indicate that thesecond-direction compression-noise estimate may be ignored. In someembodiments of the present invention, an “ignore-estimate” value used toindicate that the first-direction compression-noise estimate may beignored may be the same as an “ignore-estimate” value used to indicatethat the second-direction compression-noise estimate may be ignored. Insome embodiments of the present invention, an “ignore-estimate” valueused to indicate that the first-direction compression-noise estimate maybe ignored may be different than an “ignore-estimate” value used toindicate that the second-direction compression-noise estimate may beignored.

If the first-direction no-compression criterion is not met 640, thenstatistics may be computed 642 for the first-direction statisticalfeature values. If the second-direction no-compression criterion is notmet 650, then statistics may be computed 652 for the second-directionstatistical feature values.

In some exemplary embodiments of the present invention, the average ofthe horizontal statistical feature values and the maximum horizontalstatistical feature value may be computed, and the average of thevertical statistical feature values and the maximum vertical statisticalfeature value may be computed. These values may be computed accordingto:

$\begin{matrix}{{SF}_{H\_ {MEAN}} = {\sum\limits_{i = 0}^{{NH} - 1}\; \frac{{SF}_{H}\lbrack i\rbrack}{NH}}} \\{{{SF}_{H\_ {MAX}} = {\max\limits_{i}\left( {{SF}_{H}\lbrack i\rbrack} \right)}},} \\{i = {{0\mspace{14mu} \ldots \mspace{14mu} {NH}} - 1}} \\{{SF}_{V\_ {MEAN}} = {\sum\limits_{i = 0}^{{NV} - 1}\; \frac{{SF}_{V}\lbrack i\rbrack}{NV}}} \\{{{SF}_{V\_ {MAX}} = {\max\limits_{i}\left( {{SF}_{V}\lbrack i\rbrack} \right)}},} \\{{i = {{0\mspace{14mu} \ldots \mspace{14mu} {NV}} - 1}},}\end{matrix}$

respectively, wherein NH and NV are the number of horizontal andvertical offsets, respectively. In alternative embodiments, otherstatistical values may be calculated, for example, the median, the skew,the kurtosis and other statistical measures.

The computed statistics associated with the first-direction statisticalfeature values may be combined 644 to form an estimate of thefirst-direction compression noise. The computed statistics associatedwith the second-direction statistical feature values may be combined 654to form an estimate of the second-direction compression noise.

In an exemplary embodiment of the present invention, an estimate of thehorizontal compression noise may be calculated according to a weightedaverage given by:

Noise_(Compression) _(_) _(H)=4.64·SF _(H) _(_) _(MAX)−4.26·SF _(H) _(_)_(MEAN)+0.58,

and an estimate of the vertical compression noise may be calculatedaccording to a weighted average given by:

Noise_(Compression) _(_) _(V)=4.64·SF _(V) _(_) _(MAX)−4.26·SF _(V) _(_)_(MEAN)+0.58,

where the values 4.64, 4.26 and 0.58 are exemplary weighting parameters.In alternative embodiments, other weighting values may be used.

For example, in an alternative exemplary embodiment of the presentinvention, an estimate of the horizontal compression noise may becalculated according to a weighted average given by:

Noise_(Compression) _(_) _(H)=1.6005·SF _(H) _(_) _(MAX)−1.0764·SF _(H)_(_) _(MEAN)+2.7259,

and an estimate of the vertical compression noise may be calculatedaccording to a weighted average given by:

Noise_(Compression) _(_) _(V)=1.6005·SF _(V) _(_) _(MAX)−1.0764·SF _(V)_(_) _(MEAN)+2.7259.

The estimate for the first-direction compression noise and the estimatefor the second-direction compression noise may be combined 660 to form asingle compression noise estimate. Any data fusion method known in theart may be used to combine the estimate for the first-directioncompression noise and the estimate for the second-direction compressionnoise.

In some exemplary embodiments of the present invention, thecompression-noise estimate may be determined according to:

Noise_(Compression)=max(Noise_(Compression) _(_)_(H),Noise_(Compression) _(_) _(V)).

In an exemplary embodiment of the present invention, a first-directionnear-peak-value feature value, NearPeakD1, may be compared to afirst-direction near-peak-value threshold, and if the first-directionnear-peak-value feature value is above the first-directionnear-peak-value threshold, then the first-direction near-peak-valuefeature value may indicate an image, or video frame, uncompressed in thefirst direction. In some exemplary embodiments, the first-directionnear-peak-value threshold may be 50% of the number of first-directionoffsets. In some exemplary embodiments wherein the statistical featurevalue is the standard-deviation value, if NearPeakD1>2, then nocompression in the first direction may be indicated. In an exemplaryembodiment of the present invention, a second-direction near-peak-valuefeature value, NearPeakD2, may be compared to a second-directionnear-peak-value threshold, and if the second-direction near-peak-valuefeature value is above the second-direction near-peak-value threshold,then the second-direction near-peak-value feature value may indicate animage, or video frame, uncompressed in the second direction. In someexemplary embodiments, the second-direction near-peak-value thresholdmay be 50% of the number of second-direction offsets. In some exemplaryembodiments wherein the statistical feature value is thestandard-deviation value, if NearPeakD2>2, then no compression in thesecond direction may be indicated.

In an exemplary embodiment of the present invention, a first-directionvariance feature value, VAR_(D1), may be compared to a first-directionvariance threshold, and if VAR_(D1) is below the first-directionvariance threshold, then the first-direction variance feature value mayindicate an image, or video frame, uncompressed in the first direction.In some exemplary embodiments wherein the statistical feature value isthe standard-deviation value, if VAR_(D1)<0.6, then no compression inthe first direction may be indicated.

In an exemplary embodiment of the present invention, a second-directionvariance feature value, VAR_(D2), may be compared to a second-directionvariance threshold, and if VAR_(D2) is below the second-directionvariance threshold, then the second-direction variance feature value mayindicate an image, or video frame, uncompressed in the second direction.In some exemplary embodiments wherein the statistical feature value isthe standard-deviation value, if VAR_(D2)<0.6, then no compression inthe second direction may be indicated.

In an exemplary embodiment of the present invention, a first-directionblock-boundaries feature value, NumBlockLocsD1, may be compared to afirst-direction block-boundaries threshold, and if NumBlockLocsD1 isabove the first-direction block-boundaries threshold, then thefirst-direction block-boundaries feature value may indicate an image, orvideo frame, uncompressed in the first direction. In some exemplaryembodiments wherein the statistical feature value is thestandard-deviation value, if NumBlockLocsD1>2, when five temporal framesare examined, then no compression in the first direction may beindicated.

In an exemplary embodiment of the present invention, a second-directionblock-boundaries feature value, NumBlockLocsD2, may be compared to asecond-direction block-boundaries threshold, and if NumBlockLocsD2 isabove the second-direction block-boundaries threshold, then thesecond-direction block-boundaries feature value may indicate an image,or video frame, uncompressed in the second direction. In some exemplaryembodiments wherein the statistical feature value is thestandard-deviation value, if NumBlockLocsD2>2, when five temporal framesare examined, then no compression in the second direction may beindicated.

In an exemplary embodiment of the present invention, if the receivedadditive noise estimate is labeled a reliable estimate, then theindication is uncompressed image, or video frame.

In an exemplary embodiment of the present invention, if the receivedadditive noise estimate is greater than an additive-noise-estimatethreshold, then the indication is uncompressed image, or video frame.

In some exemplary embodiments of the present invention, when afirst-direction near-peak-value feature value indicates nofirst-direction compression OR a first-direction variance feature valueindicates no first-direction compression OR a first-directionblock-boundaries feature value indicates no first-direction compressionOR the additive-noise estimate is reliable OR is greater than anassociated threshold value, then the first-direction compression-noiseestimate may be set to an “ignore-estimate” value indicating that thefirst-direction compression-noise estimate may be ignored. Otherwise,the first-direction compression-noise estimate may be computed bycombining one, or more, statistics computed using the first-directionstandard-deviation values.

In some exemplary embodiments of the present invention, when asecond-direction near-peak-value feature value indicates nosecond-direction compression OR a second-direction variance featurevalue indicates no second-direction compression OR a second-directionblock-boundaries feature value indicates no second-direction compressionOR the additive-noise estimate is reliable OR is greater than anassociated threshold value, then the second-direction compression-noiseestimate may be set to an “ignore-estimate” value indicating that thesecond-direction compression-noise estimate may be ignored. Otherwise,the second-direction compression-noise estimate may be computed bycombining one, or more, statistics computed using the second-directionstandard-deviation values.

In some embodiments of the present invention, corresponding thresholdsin the first direction and the second direction may be the same. Inalternative embodiments, corresponding thresholds in the first directionand the second direction may be different.

Some embodiments of the present invention described herein may comprisean edge-preserving filter. In some embodiments, the edge-preservingfilter may comprise a bi-lateral filter. Alternative embodiments of thepresent invention may comprise a hierarchy, or bank, of edge-preservingfilters. In some of the embodiments of the present invention comprisinga hierarchy, or bank, of edge-preserving filters, the edge-preservingfilters may be bi-lateral filters.

FIG. 29 depicts exemplary embodiments of the present inventioncomprising a hierarchy 700 of edge-preserving filters 704, 706, 708. Insome of these exemplary embodiments, an input image 702, for example, ahigh-definition image or a 4K image, may be made available to firstedge-preserving filter 704 and a second edge-preserving filter 706. Thefiltering parameter values 705, 707 for the first edge-preserving filter704 and the second edge-preserving filter 706 may depend on noisestatistics associated with the input image 702. In some embodiments ofthe present invention, the first edge-preserving filter 704 filteringparameter value 705 may be set to a first factor of a noise estimate ofthe noise in the input image 702, and the second edge-preserving filter706 filtering parameter value 707 may be set to a second factor of thenoise estimate of the noise in the input image 702. In some embodiments,the value of the second factor may be greater than the value of thefirst factor.

A first channel 710 may be generated by taking the difference 712 of theinput image and the results 714 of the first edge-preserving filter 704.The first channel 710 may contain the image noise, and the first channel710 may be attenuated or removed 716.

The second channel 718 may be generated by taking the difference 720 ofthe results 714 of the first edge-preserving filter 704 and the results722 of the second edge-preserving filter 706. The second channel 718 maycontain a combination of image noise and texture and may be attenuated724 to reduce the noise content from the input image 702. The secondchannel 718 may not be removed completely in order to preserve the imagetexture.

A third channel 726 containing texture or significant features in theinput image 702 may be generated by filtering the output 722 of thesecond edge-preserving filter 706 using a third edge-preserving filter708 with a very high value of a filtering parameter 709 and subtracting726 the results 728 of the third edge-preserving filter 708 from thesecond-edge-preserving-filter results 722. The third channel 726 may besharpened, or enhanced, 730 to improve the details present in the inputimage 702. The results 728 of the third edge-preserving filter 708 maycontain the structure of the input image 702.

The results 732 of the noise attenuation/removal, the results 734 of theattenuation of the noise and texture data 718, the sharpened 736, orenhanced, texture and significant features 726 and the image structure728 may be combined 738 to form an enhanced image 740.

In some embodiments of the present invention, noise estimation methodsand systems described herein may be used to determine a noise estimateon which the filtering parameters 705, 707, 709 may be based.

A person of ordinary skill in the art will recognize the existence ofmany edge-preserving filters and many forms of bi-lateral filters. Aperson of ordinary skill in the art will recognize the existence of manyconfigurations of a hierarchy, or bank, of edge-preserving filters forseparation of an input image into a plurality of channels. The exemplaryfilters described herein are for purposes of illustration and are notintended to limit the scope of the present invention,

Some embodiments of the present invention may comprise a computerprogram product comprising a non-transitory, computer-readable storagemedium having instructions stored thereon/in which may be used toprogram a computing system to perform any of the features and methodsdescribed herein. Exemplary computer-readable storage media may include,but are not limited to, flash memory devices, disk storage media, forexample, floppy disks, optical disks, magneto-optical disks, DigitalVersatile Discs (DVDs), Compact Discs (CDs), micro-drives and other diskstorage media, Read-Only Memory (ROMs), Programmable Read-Only Memory(PROMs), Erasable Programmable Read-Only Memory (EPROMS), ElectricallyErasable Programmable Read-Only Memory (EEPROMs), Random-Access Memory(RAMS), Video Random-Access Memory (VRAMs), Dynamic Random-Access Memory(DRAMs) and any type of media or device suitable for storinginstructions and/or data.

The terms and expressions which have been employed in the foregoingspecification are used therein as terms of description and not oflimitation, and there is no intention in the use of such terms andexpressions of excluding equivalence of the features shown and describedor portions thereof, it being recognized that the scope of the inventionis defined and limited only by the claims which follow.

What is claimed is:
 1. A method for estimating noise in an image, themethod comprising: receiving image data corresponding to a region ofpixel values defined by a height and a width; determining a number ofoffsets for the region of pixel values, wherein an offset defines asub-set of pixel locations within the region; for each offset, computingthe standard deviation of respective directional difference valuescomputed for each pixel location included in a sub-set; computing themaximum of the computed standard deviations; computing the mean of thecomputed standard deviation; and computing a noise estimate in the imagebased on a function of the computed maximum of the standard deviationsand the computed mean of the standard deviations.
 2. The method of claim1, wherein a directional difference for a pixel location is computed asthe difference between a pixel value at a current pixel location and apixel value at an adjacent pixel location.
 3. The method of claim 2,wherein an adjacent pixel location is a horizontal adjacent pixellocation.
 4. The method of claim 3, wherein an offset defines a sub-setof pixel locations as pixel locations included in every i-th column ofthe region.
 5. The method of claim 3, wherein an adjacent pixel locationis a vertical adjacent pixel location.
 6. The method of claim 5, whereinan offset defines a sub-set of pixel locations as pixel locationsincluded in every i-th row of the region.
 7. The method of claim 1,wherein the function of the maximum of the computed standard deviationsand the mean of the computed standard deviations includes a weightedaverage function.
 8. A method for estimating noise in an image, themethod comprising: receiving image data corresponding to a region ofpixel values defined by a height and a width; determining a number ofoffsets for the region of pixel values, wherein an offset defines asub-set of pixel locations within the region; for each offset, computingthe standard deviation of respective directional difference valuescomputed for each pixel location included in a sub-set, wherein adirectional difference for a pixel location is computed as thedifference between a pixel value at a current pixel location and a pixelvalue at an adjacent pixel location; computing one or more statistics ofthe computed standard deviations; and computing a noise estimate in theimage based on a function of the one or more computed statistics.
 9. Themethod of claim 8, wherein the one or more statistics includes one ormore of: a mean, a maximum, a median, a skew, and a kurtosis.
 10. Themethod of claim 8, wherein an adjacent pixel location is a horizontaladjacent pixel location.
 11. The method of claim 10, wherein an offsetdefines a sub-set of pixel locations as pixel locations included inevery i-th column of the region.
 12. The method of claim 8, wherein anadjacent pixel location is a vertical adjacent pixel location.
 13. Themethod of claim 10, wherein an offset defines a sub-set of pixellocations as pixel locations included in every i-th row of the region.14. An apparatus for estimating noise in an image, the apparatuscomprising: an image receiver for receiving image data corresponding toa region of pixel values defined by a height and a width; an offsetdetermination unit for determining a number of offsets for the region ofpixel values, wherein an offset defines a sub-set of pixel locationswithin the region; a directional difference calculator for computing thestandard deviation of respective directional difference values computedfor each pixel location included in a sub-set for each offset; astatistic calculator for computing one or more statistics of thecomputed standard deviations; and a noise estimator for computing anoise estimate in the image based on a function of the one or morecomputed statistics.
 15. The apparatus of claim 14, wherein the one ormore statistics includes one or more of: a mean, a maximum, a median, askew, and a kurtosis.
 16. The apparatus of claim 14, wherein adirectional difference for a pixel location is computed as thedifference between a pixel value at a current pixel location and a pixelvalue at an adjacent pixel location
 17. The apparatus of claim 16,wherein an adjacent pixel location is a horizontal adjacent pixellocation.
 18. The apparatus of claim 17, wherein an offset defines asub-set of pixel locations as pixel locations included in every i-thcolumn of the region.
 19. The apparatus of claim 16, wherein an adjacentpixel location is a vertical adjacent pixel location.
 20. The apparatusof claim 19, wherein an offset defines a sub-set of pixel locations aspixel locations included in every i-th row of the region.